Natural language processing (NLP) and natural language understanding (NLU) involve using computer processing to extract meaningful information from natural language inputs (e.g., spoken or text-based). In particular, speech recognition may be employed to determine textual representations of spoken natural language inputs. For example, a user may speak a command or query, and a speech recognition system may transcribe the spoken input. One or more named entities may be detected from the transcription of the command or query. Named entities may comprise one or more elements in a stream of text that correspond to predefined categories (e.g., names of persons, organizations, locations, books, television shows, movie titles, and the like). Named entities may be important for speech recognition in order to process a user's intent behind a spoken command or query.
Various challenges may arise in developing speech recognition systems with increased accuracy in recognizing named entities in speech. In some cases, named entities in a native language may include one or more words in a foreign language. For example, English words may be intermixed with German words in a German named entity.
Conventional speech recognition and natural language processing systems might not be able to detect different combinations of languages or other instances of code switching. Furthermore, there may be issues in distinguishing between words that have conflicting meanings in different languages. For example, a word in a foreign language may be spelled the same as a word in a native language, but both words may have different meanings and pronunciations in their respective languages. Thus, it is difficult to detect foreign words in a native language input using conventional speech recognition and natural language processing systems.